Virtual interviewing for graduate medical education recruitment and selection: A BEME systematic review: BEME Guide No. 80.

BACKGROUND: The COVID-19 pandemic caused graduate medical education (GME) programs to pivot to virtual interviews (VIs) for recruitment and selection. This systematic review synthesizes the rapidly expanding evidence base on VIs, providing insights into preferred formats, strengths, and weaknesses. METHODS: PubMed/MEDLINE, Scopus, ERIC, PsycINFO, MedEdPublish, and Google Scholar were searched from 1 January 2012 to 21 February 2022. Two authors independently screened titles, abstracts, full texts, performed data extraction, and assessed risk of bias using the Medical Education Research Quality Instrument. Findings were reported according to Best Evidence in Medical Education guidance. RESULTS: One hundred ten studies were included. The majority (97%) were from North America. Fourteen were conducted before COVID-19 and 96 during the pandemic. Studies involved both medical students applying to residencies (61%) and residents applying to fellowships (39%). Surgical specialties were more represented than other specialties. Applicants preferred VI days that lasted 4-6 h, with three to five individual interviews (15-20 min each), with virtual tours and opportunities to connect with current faculty and trainees. Satisfaction with VIs was high, though both applicants and programs found VIs inferior to in-person interviews for assessing 'fit.' Confidence in ranking applicants and programs was decreased. Stakeholders universally noted significant cost and time savings with VIs, as well as equity gains and reduced carbon footprint due to eliminating travel. CONCLUSIONS: The use of VIs for GME recruitment and selection has accelerated rapidly. The findings of this review offer early insights that can guide future practice, policy, and research.

Validity evidence for an instrument for cognitive load for virtual didactic sessions.

BACKGROUND: COVID necessitated the shift to virtual resident instruction. The challenge of learning via virtual modalities has the potential to increase cognitive load. It is important for educators to reduce cognitive load to optimize learning, yet there are few available tools to measure cognitive load. The objective of this study is to identify and provide validity evidence following Messicks' framework for an instrument to evaluate cognitive load in virtual emergency medicine didactic sessions. METHODS: This study followed Messicks' framework for validity including content, response process, internal structure, and relationship to other variables. Content validity evidence included: (1) engagement of reference librarian and literature review of existing instruments; (2) engagement of experts in cognitive load, and relevant stakeholders to review the literature and choose an instrument appropriate to measure cognitive load in EM didactic presentations. Response process validity was gathered using the format and anchors of instruments with previous validity evidence and piloting amongst the author group. A lecture was provided by one faculty to four residency programs via ZoomTM. Afterwards, residents completed the cognitive load instrument. Descriptive statistics were collected; Cronbach's alpha assessed internal consistency of the instrument; and correlation for relationship to other variables (quality of lecture). RESULTS: The 10-item Leppink Cognitive Load instrument was selected with attention to content and response process validity evidence. Internal structure of the instrument was good (Cronbach's alpha = 0.80). Subscales performed well-intrinsic load (α = 0.96, excellent), extrinsic load (α = 0.89, good), and germane load (α = 0.97, excellent). Five of the items were correlated with overall quality of lecture (p < 0.05). CONCLUSIONS: The 10-item Cognitive Load instrument demonstrated good validity evidence to measure cognitive load and the subdomains of intrinsic, extraneous, and germane load. This instrument can be used to provide feedback to presenters to improve the cognitive load of their presentations.

Emergency Medicine Virtual Conference Participants' Engagement with Competing Activities.

INTRODUCTION: Residency didactic conferences transitioned to a virtual format during the COVID-19 pandemic. This format creates questions about effective educational practices, which depend on learner engagement. In this study we sought to characterize the competitive demands for learner attention during virtual didactics and to pilot methodology for future studies. METHODS: This was a prospective, observational, cohort study of attendees at virtual didactics from a single emergency medicine residency, which employed a self-report strategy informed by validated classroom assessments of student engagement. We deployed an online, two-question survey polling across six conference days using random signaled sampling. Participants reported all activities during the preceding five minutes. RESULTS: There were 1303 responses over 40 survey deployments across six nonadjacent days. Respondents were residents (63.4%); faculty (27.5%); fellows (2.3%); students (2%); and others (4.8%). Across all responses, about 85% indicated engagement in the virtual conference within the last five minutes of the polls. The average number of activities engaged in was 2.0 (standard deviation = 1.1). Additional activities included education-related (34.2%), work-related (21.1%), social (18.8%), personal (14.6%), self-care (13.4%), and entertainment (4.4%). CONCLUSION: Learners engage in a variety of activities during virtual didactics. Engagement appears to fluctuate temporally, which may inform teaching strategies. This information may also provide unique instructor feedback. This pilot study demonstrates methodology for future studies of conference engagement and learning outcomes.

Online learning developments in undergraduate medical education in response to the COVID-19 pandemic: A BEME systematic review: BEME Guide No. 69.

BACKGROUND: The COVID-19 pandemic spurred an abrupt transition away from in-person educational activities. This systematic review investigated the pivot to online learning for nonclinical undergraduate medical education (UGME) activities and explored descriptions of educational offerings deployed, their impact, and lessons learned. METHODS: The authors systematically searched four online databases and conducted a manual electronic search of MedEdPublish up to December 21, 2020. Two authors independently screened titles, abstracts and full texts, performed data extraction and assessed risk of bias. A third author resolved discrepancies. Findings were reported in accordance with the STORIES (STructured apprOach to the Reporting in healthcare education of Evidence Synthesis) statement and BEME guidance. RESULTS: Fifty-six articles were included. The majority (n = 41) described the rapid transition of existing offerings to online formats, whereas fewer (n = 15) described novel activities. The majority (n = 27) included a combination of synchronous and asynchronous components. Didactics (n = 40) and small groups (n = 26) were the most common instructional methods. Teachers largely integrated technology to replace and amplify rather than transform learning, though learner engagement was often interactive. Thematic analysis revealed unique challenges of online learning, as well as exemplary practices. The quality of study designs and reporting was modest, with underpinning theory at highest risk of bias. Virtually all studies (n = 54) assessed reaction/satisfaction, fewer than half (n = 23) assessed changes in attitudes, knowledge or skills, and none assessed behavioral, organizational or patient outcomes. CONCLUSIONS: UGME educators successfully transitioned face-to-face instructional methods online and implemented novel solutions during the COVID-19 pandemic. Although technology's potential to transform teaching is not yet fully realized, the use of synchronous and asynchronous formats encouraged virtual engagement, while offering flexible, self-directed learning. As we transition from emergency remote learning to a post-pandemic world, educators must underpin new developments with theory, report additional outcomes and provide details that support replication.

Impact of the COVID-19 Pandemic on the Clinical Learning Environment: Addressing Identified Gaps and Seizing Opportunities.

The clinical learning environment (CLE) encompasses the learner's personal characteristics and experiences, social relationships, organizational culture, and the institution's physical and virtual infrastructure. During the COVID-19 pandemic, all 4 of these parts of the CLE have undergone a massive and rapid disruption. Personal and social communications have been limited to virtual interactions or shifted to unfamiliar clinical spaces because of redeployment. Rapid changes to the organizational culture required prompt adaptations from learners and educators in their complex organizational systems yet caused increased confusion and anxiety among them. A traditional reliance on a physical infrastructure for classical educational practices in the CLE was challenged when all institutions had to undergo a major transition to a virtual learning environment. However, disruptions spurred exciting innovations in the CLE. An entire cohort of physicians and learners underwent swift adjustments in their personal and professional development and identity as they rose to meet the clinical and educational challenges they faced due to COVID-19. Social networks and collaborations were expanded beyond traditional institutional walls and previously held international boundaries within multiple specialties. Specific aspects of the organizational and educational culture, including epidemiology, public health, and medical ethics, were brought to the forefront in health professions education, while the physical learning environment underwent a rapid transition to a virtual learning space. As health professions education continues in the era of COVID-19 and into a new era, educators must take advantage of these dynamic systems to identify additional gaps and implement meaningful change. In this article, health professions educators and learners from multiple institutions and specialties discuss the gaps and weaknesses exposed, opportunities revealed, and strategies developed for optimizing the CLE in the post-COVID-19 world.

Website usability analysis of United States emergency medicine residencies.

OBJECTIVES: The Council of Residency Directors (CORD) in Emergency Medicine (EM) has recommended that all residency programs should conduct virtual interviews for the 2020 to 2021 application cycle due to the COVID-19 pandemic. While factors such as geographical region, city, program size, or hospital affiliation are not modifiable, EM residencies can bridge the information gap created by a lack of face-to-face interaction by representing themselves digitally. Measuring usability provides an objective method for EM residencies to improve their Web presence and effectively represent themselves to applicants. METHODS: Our sample set included 55 U.S. EM residency program websites. Using methodology replicated from previous literature on health care website usability, we divided usability into four categories for quantifiable analysis: accessibility, marketing, content quality, and technology. Analysis was performed on each website and scored in all four categories. A "general usability" score was calculated for each website using a composite of the key factors within the four categories. Using a weighted percentage across all of the factors, an overall score was calculated. RESULTS: Content quality was the overall highest scoring category (mean ± SD = 5.4, SE = 0.33). The overall lowest performing category was technology (mean ± SD = 0.8 ± 0.09, SE = 0.01). CONCLUSIONS: Measuring usability can help EM residency programs identify ways to improve their Web presence. To effectively promote their programs, residencies need quality content that communicates their key features. Our recommendation is for all residency programs to periodically perform website audits and apply the usability measures outlined to improve their digital presence, especially during times when face-to-face interactions will be limited.

ALiEM Connect: Large-Scale, Interactive, Virtual Residency Programming in Response to COVID-19.

PROBLEM: The COVID-19 pandemic restricted in-person gatherings, including residency conferences. The pressure to quickly reorganize educational conferences and convert content to a remote format overwhelmed many programs. This article describes the pilot event of a large-scale, interactive, virtual educational conference modeled, designed, and implemented by Academic Life in Emergency Medicine (ALiEM), called ALiEM Connect. APPROACH: The pilot ALiEM Connect event was conceptualized and implemented within a 2-week period in March 2020. The pilot was livestreamed via a combination of Zoom and YouTube and was archived by YouTube. Slack was used as a backchannel to allow interaction with other participants and engagement with the speakers (via moderators who posed questions from the backchannel to the speakers live during the videoconference). OUTCOMES: The RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework was used for program evaluation, showing that 64 U.S. Accreditation Council for Graduate Medical Education-accredited emergency medicine residency programs participated in the pilot event, with 1,178 unique users during the event (reach). For effectiveness, 93% (139/149) of trainees reported the pilot as enjoyable and 85% (126/149) reported it was equivalent to or better than their usual academic proceedings. Adoption for ALiEM Connect was fairly good with 64/237 (27%) of invited residency programs registering and participating in the pilot event. Implementation was demonstrated by nearly half of the livestream viewers (47%, 553/1,178) interacting in the backchannel discussion, sending a total of 4,128 messages in the first 4 hours. NEXT STEPS: The final component of the RE-AIM framework, maintenance, will take more time to evaluate. Further study is required to measure the educational impact of events like the ALiEM Connect pilot. The ALiEM Connect model could potentially be used to replace educational conferences that have been canceled or to implement and/or augment a large-scale, shared curriculum among residency programs in the future.

An update on developments in medical education in response to the COVID-19 pandemic: A BEME scoping review: BEME Guide No. 64.

BACKGROUND: COVID-19 has fundamentally altered how education is delivered. Gordon et al. previously conducted a review of medical education developments in response to COVID-19; however, the field has rapidly evolved in the ensuing months. This scoping review aims to map the extent, range and nature of subsequent developments, summarizing the expanding evidence base and identifying areas for future research. METHODS: The authors followed the five stages of a scoping review outlined by Arskey and O'Malley. Four online databases and MedEdPublish were searched. Two authors independently screened titles, abstracts and full texts. Included articles described developments in medical education deployed in response to COVID-19 and reported outcomes. Data extraction was completed by two authors and synthesized into a variety of maps and charts. RESULTS: One hundred twenty-seven articles were included: 104 were from North America, Asia and Europe; 51 were undergraduate, 41 graduate, 22 continuing medical education, and 13 mixed; 35 were implemented by universities, 75 by academic hospitals, and 17 by organizations or collaborations. The focus of developments included pivoting to online learning (n = 58), simulation (n = 24), assessment (n = 11), well-being (n = 8), telehealth (n = 5), clinical service reconfigurations (n = 4), interviews (n = 4), service provision (n = 2), faculty development (n = 2) and other (n = 9). The most common Kirkpatrick outcome reported was Level 1, however, a number of studies reported 2a or 2b. A few described Levels 3, 4a, 4b or other outcomes (e.g. quality improvement). CONCLUSIONS: This scoping review mapped the available literature on developments in medical education in response to COVID-19, summarizing developments and outcomes to serve as a guide for future work. The review highlighted areas of relative strength, as well as several gaps. Numerous articles have been written about remote learning and simulation and these areas are ripe for full systematic reviews. Telehealth, interviews and faculty development were lacking and need urgent attention.